Synthetic workload generation for benchmarking LLM inference
Which open-source tools and datasets can generate synthetic workloads — chat, agentic, and other — for benchmarking LLM inference performance?
A practical map of the tools that generate workloads for benchmarking LLM inference: engine-native load generators (vLLM, SGLang, llama.cpp), trace-driven generators (BurstGPT, Azure traces, ShareGPT/LMSYS replay), statistical generators calibrated to production traffic (ServeGen, FineServe), simulators (LLMServingSim, Frontier, SimLLM), and the emerging agentic-workload tooling (XPerf, SGLang agentic-trace, GuideLLM tool calling). The evidence says workload realism matters — Poisson arrivals and fixed token lengths misrepresent production traffic — and that agentic workloads are the least standardized but fastest-moving category. Confidence is moderate: repo capabilities were verified from primary docs fetched this session, but several tools change weekly and one major tool (NVIDIA GenAI-Perf) was unretrievable.
Updated 1 Sept 202626 sources2019–2026Standard15 min read
LLM inference · benchmarking · synthetic workloads · serving · agentic workloads · load generation